Over-optimistic evaluation and reporting of novel cluster algorithms: an illustrative study

Abstract When researchers publish new cluster algorithms, they usually demonstrate the strengths of their novel approaches by comparing algorithms’ performance with existing competitors. However, such studies are likely to be optimistically biased towards as authors have a vested interest in presenting method favorably possible order increase chances getting published. Therefore, superior newly introduced algorithms is over-optimistic and might not confirmed independent benchmark performed neutral unbiased authors. This problem known among many researchers, but so far, different mechanisms leading over-optimism algorithm evaluation never been systematically studied discussed. Researchers thus often aware full extent problem. We present an illustrative study illuminate which authors—consciously or unconsciously—paint algorithm’s light. Using recently published Rock example, we how optimization used datasets data characteristics, parameters choice competing leads Rock’s appearing better than it actually is. Our cautionary tale that illustrates easy can for claim apparent “superiority” algorithm. illuminates vital importance strategies avoiding problems (such as, e.g., studies), also discuss article.